System and method for using an event window for testing an event processing system

ABSTRACT

Systems and methods can use a testing framework for testing an event processing system. The testing framework operates to send one or more input events in an event stream to an event processing system, wherein each said input event is associated with a timestamp. Also, the testing framework can receive one or more actual output events from the event processing system, wherein each said actual output event is associated with a timestamp. Furthermore, the testing framework can compare said one or more actual output events in an event window with one or more expected output events for the event window.

CLAIM OF PRIORITY

This application claims priority on U.S. Provisional Patent ApplicationNo. 61/971,283, filed Mar. 27, 2014 entitled “XML-BASED TESTINGFRAMEWORK FOR EVENT PROCESSING LANGUAGE” (Attorney Docket No.ORACL-05525US0) which application is herein incorporated by reference inits entirety.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to the following patent applications, eachof which is hereby incorporated by reference in its entirety:

U.S. patent application Ser. No. 12/133,220, filed Jun. 4, 2008 entitled“XML-BASED EVENT PROCESSING NETWORKS FOR EVENT SERVER” (Attorney DocketNo. ORACL-02256US3).

U.S. patent application Ser. No. ______, filed February 2015, entitled“SYSTEM AND METHOD FOR SUPPORTING A SLIDING WINDOW FOR TESTING AN EVENTPROCESSING SYSTEM” (Attorney Docket No. ORACL-05525U52);

U.S. patent application Ser. No. ______, filed February 2015, entitled“SYSTEM AND METHOD FOR CONFIGURING A SLIDING WINDOW FOR TESTING AN EVENTPROCESSING SYSTEM BASED ON A SYSTEM TIME” (Attorney Docket No.ORACL-05525U53); and

U.S. patent application Ser. No. ______, filed February 2015, entitled“SYSTEM AND METHOD FOR TESTING AN EVENT PROCESSING SYSTEM WITH MULTIPLEINPUT EVENT STREAMS” (Attorney Docket No. ORACL-05525US4).

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightrights whatsoever.

FIELD OF INVENTION

The present invention is generally related to information technology,and particularly related to testing an event processing system.

BACKGROUND

In modern society, the ability to process a large amount of data in realtime becomes a crucial competitive fact for a company. For example, alarge number of sensors and smart devices, which are connected to a fastnetwork, can be used concurrently. An event processing system can beused for processing the data generated from these sources, which areinherently streaming data sources. It is desirable to automate the testcases for the event processing system. These are the generally areasthat embodiments of the invention are intended to address.

SUMMARY

Described herein are systems and methods that can use a testingframework for testing an event processing system. The testing frameworkoperates to send one or more input events in an event stream to an eventprocessing system, wherein each said input event is associated with atimestamp. Also, the testing framework can receive one or more actualoutput events from the event processing system, wherein each said actualoutput event is associated with a timestamp. Furthermore, the testingframework can compare said one or more actual output events in an eventwindow with one or more expected output events for the event window.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 shows an illustration of a framework for testing an eventprocessing system, in accordance with an embodiment of the invention.

FIG. 2 shows an illustration of an XML test file for an event processingsystem, in accordance with an embodiment of the invention.

FIG. 3 shows an illustration of a testing framework based on aclient/server architecture, in accordance with an embodiment of theinvention.

FIG. 4 shows an illustration of testing an event processing system usingdifferent strategies for input and verification, in accordance with anembodiment of the invention.

FIG. 5 illustrates an exemplary flow chart for testing an eventprocessing system using different strategies for input and verification,in accordance with an embodiment of the invention.

FIG. 6 shows an illustration of using an event window for testing anevent processing system, in accordance with an embodiment of theinvention.

FIG. 7 shows an illustration of verifying the outputs of a query fortesting an event processing system, in accordance with an embodiment ofthe invention.

FIG. 8 shows an illustration of testing an event processing system basedon a system time, in accordance with an embodiment of the invention.

FIG. 9 shows an illustration of testing an event processing system basedon an application time, in accordance with an embodiment of theinvention.

FIG. 10 illustrates an exemplary flow chart for using an event windowfor testing an event processing system, in accordance with an embodimentof the invention.

FIG. 11 shows an illustration of supporting a sliding window for testingan event processing system based on an application time, in accordancewith an embodiment of the invention.

FIG. 12 shows an illustration of verifying the outputs of the query witha system timestamp for testing an event processing system, in accordancewith an embodiment of the invention.

FIG. 13 shows an illustration of testing an event processing systemusing a sliding window based on an application time, in accordance withan embodiment of the invention.

FIG. 14 illustrates an exemplary flow chart for supporting a slidingwindow for testing an event processing system based on an applicationtime, in accordance with an embodiment of the invention.

FIG. 15 shows an illustration of supporting a sliding window for testingan event processing system based on a system time, in accordance with anembodiment of the invention.

FIG. 16 shows an illustration of testing an event processing systemusing a sliding window based on a system time, in accordance with anembodiment of the invention.

FIG. 17 illustrates an exemplary flow chart for supporting a slidingwindow for testing an event processing system based on a system time, inaccordance with an embodiment of the invention.

FIG. 18 shows an illustration of supporting testing an event processingsystem with multiple input event streams, in accordance with anembodiment of the invention.

FIG. 19 shows an illustration of using multiple event windows fortesting an event processing system, in accordance with an embodiment ofthe invention.

FIG. 20 illustrates an exemplary flow chart for supporting testing anevent processing system with multiple input event streams, in accordancewith an embodiment of the invention.

DETAILED DESCRIPTION

The invention is illustrated, by way of example and not by way oflimitation, in the figures of the accompanying drawings in which likereferences indicate similar elements. It should be noted that referencesto “an” or “one” or “some” embodiment(s) in this disclosure are notnecessarily to the same embodiment, and such references mean at leastone.

The description of the invention as following uses the continuous querylanguage (CQL) as an example for an event processing language. It willbe apparent to those skilled in the art that other types of eventprocessing languages can be used without limitation.

Described herein are systems and methods that can support a testingframework for an event processing system.

A Testing Framework for Event Stream Processing

In an event processing system (a.k.a. an event stream processingsystem), an event indicates a change in the real world that can berepresented or generated digitally using a computing system.Furthermore, an event stream is a sequence of events that are generatedcontinuously. For example, the order of the events in the event streammay be the order that the events are generated or appended.

The event processing system, such as the Oracle Complex Event Processing(CEP) system, can take advantage of a real-time event drivenarchitecture that is capable of processing a large number of continuousdata with low latency, e.g. using filtering, aggregation, correlation,pattern match, trend detection and other techniques, before publishingthe output events.

The event processing system can be based on different event processinglanguages, which can be more complex than other data processinglanguages. Also, the different event processing languages may sharecommon concepts, functionalities and structures (such as WHERE, GROUPBY, HAVING, ORDER BY, JOIN, and collection operations). For example, anevent processing language (e.g. the CQL) can be derived from thestructured query language (SQL). Also, the event processing language canincorporate various event stream processing specific concepts, such asthe event window, pattern match, stream-relation conversion operators,into the SQL.

In accordance with an embodiment of the invention, an event processingsystem can read and process the input data, and outputs the result basedon various rules. The rules can be stored in plain text file, which iseasy for humans to read and edit. Also, modifications can be made overthe network and can take effect immediately.

Furthermore, the event processing system can extend the rule language tomeet various domain specific requirements. Thus, the users can focus onthe business logic rather than the complex technical procedures.

FIG. 1 shows an illustration of a framework for testing an eventprocessing system, in accordance with an embodiment of the invention. Asshown in FIG. 1, a testing environment 100 can support a testingframework 101, which is capable of testing an event processing system102.

In accordance with an embodiment of the invention, the testing framework101 supports a declarative approach for automating various tests. Forexample, the different testing steps can be automated using variousdeclarative scripts, such as the Ant scripts, which can be imported intothe testing framework 101.

As shown in FIG. 1, the testing framework 101 can be based on an XMLschema 112, which is designed for testing an event processing languagethat is used in the event processing system 102. The event processinglanguage can have complex structures, different operators and functions.For example, a query in the event processing language may have a largenumber of forms. Also, a query may require different input data to coverdifferent scenarios or paths.

Furthermore, the XML schema 112 can declare the data structures, rules,input data, expected output data, and different verification logics fortesting the event processing system 102.

The testing framework 101 can use an XML file 111 for specifying theinput data, the processing queries, and the expected output in theformat as declared in the XML schema 112. Also, the testing framework101 allows the users to organize the tests in a tree structure and canshare the input data and event types across different test cases.

In accordance with an embodiment of the invention, the testing framework101 can parse the XML test file 111 and locate the tests (e.g. per therequest from a client). For example, before executing the tests in thetest XML file 111, the testing framework 101 can convert the tests intoan event processing network (EPN) that the concrete event streamprocessing system 102 supports.

Also, the testing framework 101 can conveniently verify the results ofthe event stream processing system 102. For example, the testers (or thedevelopers and the users of the event stream processing system 102) canadd or modify one or more tests by editing the XML file 111, in whichcase the testing framework 101 is able to execute the tests withoutcompilation.

Thus, the testing framework 101 allows the test developers to easilyhandle the input data, queries, and expected output in the tests.

FIG. 2 shows an illustration of an XML test file for an event processingsystem, in accordance with an embodiment of the invention. As shown inFIG. 2, a testing framework can organize various tests in an XML testfile 200 based on a tree structure.

The top level (e.g., a root element) of the XML test file is atest-suite element 201 that defines a test suite (e.g. a <test-suite/>element in XML), which may represent a set of tests for testing aparticular feature of an event processing language that is used in anevent processing system.

As shown in FIG. 2, the test-suite element 201 can comprise one or moreevent type definitions 202 (such as the event types 1-2 208-209) andvarious test cases 203-204. Each event type represents a data structurethat may comprise a number of properties, such as the name, time,location, measures, and associations with other events, which providesvarious context information for an event.

The following is an exemplary schema definition for a <test-suite/>element in XML.

<xsd:element name=″test-suite″ type=″TestSuiteType″ /> <xsd:complexTypename=″TestSuiteType″>    <xsd:sequence>       <xsd:elementname=″event-type-repository″    minOccurs=”1” maxOccurs=″1″   type=″EventTypeRepositoryType″/>       <xsd:element name=″test-case″minOccurs=”1”    maxOccurs=″unbounded″ type=″TestCaseType″ />   </xsd:sequence>    <xsd:attribute name=″name″ type=″xsd:string″use=″required″ /> </xsd:complexType>

As shown in the above, the <test-suite/> element in XML comprises one<event-type repository/> element, which defines various event types, andone or multiple <test-case/> elements, which define one or more testcases.

In accordance with an embodiment of the invention, the testing framework200 can support different kinds of event types, such as a class basedevent type and a map based event type, for event stream processing.

The class based event type can be a binary stereotyped object, which isbound to a specific programming language and can be reused in differenttest suites. For example in Java, the class based event type can be aJava bean class.

The following is an exemplary schema definition for a class based eventtype.

<xsd:complexType name=“EventTypeType”>    <xsd:sequence minOccurs=“0”>      <xsd:choice minOccurs=“0”>          <xsd:element name=“properties”         type=“PropertyType”/>          <xsd:element name=“class”type=“xsd:string”/>       </xsd:choice>    </xsd:sequence>   <xsd:attribute name=“type-name” type=“xsd:string”    use=“required”/></xsd:complexType>

The map based event type can be declared in the XML test file byspecifying the properties and data types of an event. The map basedevent type supports a key-value structure that provides flexibility forthe event stream processing system to maintain additional informationthat can be useful for the system to process the events internally (suchas in the rules). For example, the additional information may includepseudo columns such as the timestamp of an event and the kind of anevent (e.g. insert/delete/update).

The following is an exemplary schema definition for a map based eventtype.

<xsd:complexType name=“PropertyType”>    <xsd:sequence>      <xsd:element name=“property” maxOccurs=“unbounded”>         <xsd:complexType>             <xsd:attribute name=“name”         type=“xsd:string” use=“required”/>             <xsd:attributename=“type”             type=“xsd:string” use=“required”/>            <xsd:attribute name=“length”             type=“xsd:int”use=“optional”/>       </xsd:complexType>       </xsd:element>   </xsd:sequence> </xsd:complexType>

In accordance with an embodiment of the invention, each of the test caseelements 203-204 can define the data model and the processing logic(e.g. representing a set of tests for a particular rule). As shown inFIG. 2, a test case element 203 (e.g. a <test-case/> element in XML)defines the input streams in the input channel element 210, defines theoutput stream in the output channel element 211, and the rules 213 inthe processor element 212. Also, a test case element 204 defines therules 215 in the processor element 214.

The following is an exemplary schema definition of a <test-case/>element in XML.

<xsd:complexType name=“TestCaseType”>    <xsd:sequence>      <xsd:element name=“input-channel” minOccurs=“1”   maxOccurs=“unbounded”    type=“EventChannelType”></xsd:element>      <xsd:element name=“output-channel” minOccurs=“0”    maxOccurs=“1”   type=“EventChannelType”></xsd:element>       <xsd:elementname=“processor” type=“ProcessorType”    minOccurs=“1” maxOccurs=“1”/>      <xsd:element name=“test” minOccurs=“1”    maxOccurs=“unbounded”type=“TestType” />    </xsd:sequence>    <xsd:attribute name=“name”type=“xsd:string” use=“required” /> </xsd:complexType>

As shown in the above, the <test-case/> element in XML comprises one ormultiple <input-channel/> element, zero or one <output-channel/>element, one <processor/> element, and one or multiple <test/> elements.

In accordance with an embodiment of the invention, an event channel is acomponent used in the event stream processing system for deliveringevents between various components. The event channel can be bound withan event type. Either an event type name or an event channel name can beused in a rule for referencing the stream of specific events. Thus, theevents can be passed from an event channel to a processor and from aprocessor to an event channel.

For example, as shown in the above, both the <input-channel/> elementand the <output-channel/> element can be bound with an event type (andare associated with the same EventChannelType). Additionally, the<processor/> element may only allow one executive rule, whereas theother rules that are used or referenced in the executive rule can serveas sub-queries.

In accordance with an embodiment of the invention, each of the testelements 205-207 can represent a concrete test and provides the inputand expected output for the test. For example, the test elements 205-207can take advantage of a data driven model, which reflects the nature ofthe streaming event system.

As shown in FIG. 2, each test case in the test suite 201 can contain anumber of tests, while each test can have one or more asserts. Forexample, the test case 203 can contain a number of tests 205-206, thetest 205 may include asserts 216-217 and the test 206 may include theassert 218. Furthermore, the test case 204 can contain a test 207, andthe test 207 may include asserts 219-220.

The following is an exemplary schema definition of a <test/> element inXML.

<xsd:complexType name=″TestType″> <xsd:sequence> <xsd:elementname=″assert″ minOccurs=″1″ maxOccurs=″unbounded″> <xsd:complexType ><xsd:choice minOccurs=″0″ maxOccurs=″unbounded″> <xsd:elementname=″input-events″ type=″InputEventsType″/> <xsd:elementname=″output-events″ type=”OutputEventsType”/> </xsd:choice><xsd:attribute name=″id″ type=″xsd:string″ use=″optional″ /></xsd:complexType> </xsd:element> </xsd:sequence> <xsd:attributename=″name″ type=″xsd:string″ use=″required″ /> </xsd:complexType>

As shown in the above, the <test/> element in XML can comprise one ormultiple <assert/> elements, and each <assert/> element comprises one ormultiple <input-events> elements and zero or one <output-events/>element.

In accordance with an embodiment of the invention, each of the assertelements 216-220 can have one or more input-events and output-events.For example, in order to simplify the testing of the event processingsystem using an event processing language, the testing framework 200 mayrequire that each assert in a test can have one or multiple input-eventsand only one output-event.

Furthermore, the flexible structure of an assert element allows a userto author tests taking advantage of different input and verificationstrategies.

In accordance with an embodiment of the invention, the events 221-228 inthe testing framework can be modeled as input-events and output-eventsrespectively. The input-events and the output-events can be bound withan event channel (i.e. with an event type). For example, if the eventsare provided directly in the XML test file, the event type name can beused as the element name and the properties can be specified as theattributes of the element. If the events are provided in an externalcomma separated values (CSV) file, the property names of the event typecan be stated at the header.

In accordance with an embodiment of the invention, the frameworksupports data sharing among tests. For example, the different eventtypes 208-209 can be shared among different test cases 203-204 in thetest suite 201. Different tests 205-206 in one test case 203 can sharethe same processing rules 213. Additionally, different tests 205-207 canshare the same input data/output data by using external files, e.g. CSVfiles. Thus, the users can design the tests 205-207 based on a uniformeddata model.

The following List 1 is an exemplary XML file that is based on the aboveschema definitions.

List 1 <event-type-repository> <event-type type-name=”StockEvent”><class>cep.tests.events.StockEvent</class> </event-type> <event-typetype-name=”TradeEvent”> <properties> <property name=”product”type=”string” length=”80”/> <property name=”price” type=”float”/><property name=”amount” type=”int”/> <property name=”tradeTime”type=”timestamp”/> </properties> </event-type> </event-type-repository><input-channel id=″s1″ event-type=″StockEvent″ /> <input-channel id=″s2″event-type=″StockEvent″ /> <output-channel id=″s3″event-type=″TradeEvent″/> <processor> <rule id=”q1” type=”query”> SELECT”STOCK” as product, s1.price as price, s2.createTime as tradeTime FROMchannel1 as s1, channel2 as s2 WHERE s1.symbol=s2.symbol AND s1.price <s2.price </rule> </processor> <assert> <input-events channel=”channel1”><StockEvent symbol=”ABC” price=”19.8” createTime =”10080900”><StockEvent symbol=”AAA” price=”29.9” createTime =”10080901”></input-events> <input-events channel=”channel2” file=”input/stock.csv”><input-events> <output-events channel=”s3” verify=”all”> <TradeEventproduct=”STOCK” price=”19.8” tradeTime=” 10080901”/> output-events></assert>

As shown in the above, the XML file defines an event type of StockEventand an event type of TradeEvent in the event type repository.Additionally, the input channels s1 and s2 are associated with theStockEvent, and the output channel s3 is associated with the TradeEvent.In the rule section, the event channel name s1 and s2 are used tospecify from which stream the input events may come.

As the test is executed, the testing framework reads in the input eventsdata, converts the input events data into the associated StockEventevent type, and sends the events to the input channels s1 and s2. Then,the rule engine in the event processing system can process the inputevents based on the logic specified in the rule section, generates a newschema for the output events using the TradeEvent event type, and sendsthe output events to the output channel s3, where an event listener isregistered by the framework to hold and send output events.

Once the input events are sent, the testing framework can read in theoutput events data from the XML file, converts the output events datainto the associated event type, TradeEvent, and compares these expectedoutput events with the actual output events that are received from theevent channel s3, as indicated by the “channel” attribute in the<output-events/> element.

FIG. 3 shows an illustration of a testing framework based on aclient/server architecture, in accordance with an embodiment of theinvention. As shown in FIG. 3, a testing environment 300 can support atesting framework 301 for testing an event processing system 302.

A test client 320 can deploy the testing framework 301 on the server310, which runs on (or is connected to) the event stream processingsystem 302. In accordance with an embodiment of the invention, thetesting framework 301 can be compiled and built before the event streamprocessing system 302 is started. Also, the event stream processingsystem 302 (or a rule engine) can be started prior to the running of thetests.

In accordance with an embodiment of the invention, the test client 320can pass various parameters to the server 310 to inform the testingframework 301 where to find the tests to be executed. In addition, thetesting framework 301 is able to load the library that defines variousdata structures used by the testing framework 301. Also, the testingframework 301 is able to know the path to find the external files thatstore the input data and expected results 314 (or data).

Furthermore, the testing framework 301 can register the data structuresand the rules 313 defined in the test file 311. Also, the testingframework 301 can create the components that are used for handling inputand output and create the component for handling the rules.Additionally, the testing framework 301 can register the customizedcomponent to receive the result from the output component.

As shown in FIG. 3, when the test client 320 invokes a test for testingthe event stream processing system 302, the testing framework 301 canread the input data 312, provides the rules 313 to the event streamprocessing system 302, collects the actual output events 304 (i.e. theactual result), and compares the actual output events 304 with theexpected results 314 (or throwing exception with the failure details toindicate a failing test).

Furthermore, when the testing framework 301 receives the actual outputevents 304, the testing framework 301 can construct an internalrepresentation for each actual output event 304. The internalrepresentation for the actual output events 304 may have the same (orsimilar) structure as an expected output event. Also, different pseudocolumn names can be used for the different event processing languages.

The following shows a structure of an exemplary internal representation.

class MyEvent { timestamp: long kind:String event:Object }

As shown in the above, timestamp and kind are pseudo properties, whichmay not be part of the event type schema as defined in the<event-type-repository/> element. Also, both the class based events andthe map based events can have the same pseudo properties, such astimestamp and kind, which can be used for both input and verification.

For example, the testing framework 301 can set the pseudo propertytimestamp according to the value of the pseudo column “timestamp” of theactual event and can set the property kind according to the pseudocolumn “kind” of the actual event, while setting the property eventdirectly according to the actual event.

Finally, after the tests are executed, the testing framework 301 cangenerate a report to present the status of the execution (or provide thedetails of the failures).

In accordance with an embodiment of the invention, the testing framework301 is capable of supporting different strategies for input andverification, providing event window with transparent timestamps,supporting input from multiple streams, verifying the result of slidingwindow based on an application timestamp, and verifying the result ofsliding window based on a system timestamp.

Different Strategies for Input and Verification

FIG. 4 shows an illustration of testing an event processing system usingdifferent strategies for input and verification, in accordance with anembodiment of the invention. As shown in FIG. 4, a test 401 in a testingframework 400 can include one or more asserts 411-412. The assert 411can define one or more input events 421 and one or more expected outputevents 431, while the assert 411 can define one or more input events 422and one or more expected output events 432.

The testing framework 400 can send the various input events to an eventprocessing system 402 using one or more pre-configured input channels403. Additionally, the testing framework 400 can use a processor 410 todefine one or more executable rules 420 for the event processing system402. For example, the testing framework 400 can define only oneexecutable rule in each test in order to simplify the testing task.Also, an executable rule can reference to other rules that may serve assub-queries.

Then, the testing framework 400 can receive one or more actual outputevents 430 from an output channel 404 configured for the eventprocessing system 402, after the event processing system 402 hasprocessed the incoming events based on the rule processing.

In accordance with an embodiment of the invention, the testing framework400 can use different verification strategies for verifying differentasserts. As shown in FIG. 4, the testing framework 400 can use differentverification strategies 441-442 for matching the received actual outputevents 430 with the expected output events 431-432.

For example, the testing framework 400 allows a user to use thefollowing verification strategies:

-   -   Input-output verification    -   Stepped input-output verification    -   Event-by-event verification    -   Prepared data in event window    -   Multiple streams input

The input-output verification strategy, which is a basic strategy,allows the <test/> element to only have one <assert/> element, andallows the <assert/> element to only have one <input-events> element andone <output-events> element. The following is an exemplary <test/>element using the input-output verification strategy.

<test id=”testFunctionMax”> <assert> <input-events channel=”s1”><MyEvent .../> <MyEvent .../> </input-events> <output-eventschannel=”s2”> <Event .../> <Event .../> </output-events> </assert></test>

The stepped input-output verification strategy is an extension to theinput-output verification strategy. Using the stepped input-outputverification strategy, an <assert/> element can include multiple<input-events/> and <output-events/> pairs. The timestamp of the eventsin an <input-events/> are successive to the timestamp of the events inthe previous <input-events/>. Thus, the user is able to test aparticular rule with different inputs from different aspects.

The following is an exemplary <test/> element using the steppedinput-output verification strategy.

<test id=” testFunctionMax”> <assert> <input-eventschannel=”s1”></input-events> <output-eventschannel=”s2”></output-events> <input-events channel=”s1”></input-events><output-events channel=”s2”></output-events> </assert> </test>

The event-by-event verification strategy can be beneficial forsupporting continuous verification. The <assert/> element can havemultiple <input-events> and <output-events> pairs, while there is onlyone event to input in each <input-events/> element. Thus, the user canensure that the insert/delete events are generated properly in theprocess.

There can also be a strategy for preparing data in an event window. Thestrategy allows the first <assert/> element, in a set of multiple<assert/> elements, to be only used for preparing the data in the eventwindow without verifying the output. As a result, the test is able tostart with non-empty event window, when there are a large number ofevents involved.

The following is an exemplary <test/> element using the strategy ofprepared data in an event window.

<test id=”testFunctionMax”> <assert> <input-eventschannel=”s1”></input-events> </assert> <assert> <input-eventschannel=”s1”></input-events> <output-eventschannel=”s2”></output-events> </assert> </test>

Also, a strategy can be used when the rule deals with multiple streams.For example, the streaming events may be inputted in parallel since thedifferent streams may send input data simultaneously. The framework caninsert the events asynchronously by using a “parallel” attribute on the<input-events/> element, while guaranteeing that the events are sent atthe specified time.

The following is an exemplary <test/> element using the strategy formultiple streams input.

<test id=”testUnionOperator”> <assert> <input-events channel=”s1”parallel=”true”></input-events> <input-events channel=”s2”parallel=”true”></input-events> <output-eventschannel=”s3”></output-events> </assert> <assert> <input-eventschannel=”s2”></input-events> <output-eventschannel=”s3”></output-events> </assert> </test>

FIG. 5 illustrates an exemplary flow chart for testing an eventprocessing system using different strategies for input and verification,in accordance with an embodiment of the invention. As shown in FIG. 5,at step 501, a testing framework can organize one or more tests in atree structure, wherein each said test includes a plurality of asserts,wherein each assert defines one or more input events and an expectedoutput event. At step 502, the testing framework can receive one or moreactual output events from the event processing system. Furthermore, atstep 503, the testing framework can use an individual verificationstrategy to verify each different said assert based on the one or moreactual output events and the expected output event

Event Window and Timestamp

In accordance with an embodiment of the invention, each event can beassociated with a time (e.g. using a timestamp). Furthermore, the eventstream processing system can use an event window, which moves forwardalong with the time, to indicate a range of events (e.g. up to thelatest events) for processing.

FIG. 6 shows an illustration of using an event window for testing anevent processing system, in accordance with an embodiment of theinvention. As shown in FIG. 6, a testing framework 600 can provide anevent stream 660 to an event processing system 620, which can processthe events in the event stream 660 (using either a per-event mode or abatch mode).

The testing framework 600 can apply an event window 610 on the eventstream 660. The event window 610 can include one or more input events601-603, which are received and processed by the event processing system620. Furthermore, each said input event 601-603 can be associated with atimestamp 611-613, the value of which may be configured in a test XMLfile.

In accordance with an embodiment of the invention, a timestamp can bebased on either a system time or an application time. For example, astream of events can be based on an application time (or be referred toas being application timestamped), if the event has an attribute holdingthe time information, the value of which can be used in the eventprocessing.

On the other hand, a stream of these events can be based on a systemtime (or be referred to as being system timestamped), if there is notime attribute in the event data structure, or such attributes are notused in event processing. In such a case, the testing system may attacha system time to the event when the event is sent to the eventprocessing system 620.

Furthermore, the testing framework 600 allows the users to design thetests transparently using both the system timestamp and the applicationtimestamp, which are logically equivalent. Also, the model of thetimestamp is transparent to the rules 640 used by the event processingsystem 620 (and/or the rule engine in the testing framework 600).

In accordance with an embodiment of the invention, the testing framework600 can use different processing logics for processing the applicationtimestamped event stream and the system timestamped event stream.

For example, the value of the timestamp property for an applicationtimestamped event stream may be considered as an absolute value, whilethe value of the timestamp property for a system timestamped eventstream may be considered as a relative value.

Furthermore, the testing framework 600 can provide special configurationfor the timestamp property associated with an application timestampedevent stream. For example, the application timestamp can be specifiedusing an attribute or an expression in a test file. Also, the testingframework 600 can employ special processes for performing tests usingthe system timestamps. For example, the testing framework 600 cancalculate a sliding point in the system time, when applying an eventwindow on a system timestamped event stream.

Additionally, the testing framework 600 supports different types ofspecial temporal events, such as a wait event for a system timestampedevent stream and a heartbeat event for an application timestamped eventstream.

Moreover, the testing framework 600 allows the timestamp to be definedin different units. For example, the timestamp unit in a test XML filecan be prescribed in milliseconds, while the timestamp used internallyin the event processing system may be in nanoseconds.

In accordance with an embodiment of the invention, the testing framework600 supports both a time based event window and an event based eventwindow (for both the per-event mode and the batch mode).

A time based event window represents a set of events that are insertedinto the event stream during a period of time as specified. For example,a time based event window can be used with a query, SELECT * FROM s1[RANGE 3 SECONDS], which indicates that the query computes the eventsfrom the current time back three (3) seconds. Using such a time basedwindow, when a new event is inserted into the event stream 660, thetesting framework 600 can add the new event in the event window 610 andoutput an insert event. When the event is older than three seconds, thetesting framework 600 can delete the event from the event window 610 andoutput a delete event.

An event based event window represents a set of events, the number ofwhich may be specified. For example, an event based event window can beused with a query, SELECT * FROM s1[ROWS 3], which indicates that thequery computes the events from the last event back to the third event inthe stream (in the reverse order). Using such an event based window,when a new event is inserted into the event stream 660, the testingframework 600 can add the new event in the event window 610 and outputsan insert event. Also, prior to the insertion, if there are more thanthree (3) events, the testing framework 600 can delete the oldestevent(s) from the event window 610 and output a delete event.

As shown in FIG. 6, the event processing system 620 can generate theoutput events 650 based on the rules processing (using the timestamp611-613 values). Also, the testing framework 600 can prescribe one ormore expected output events 630 for the input events 601-603.

Then, the testing framework 600 can use the expected output events621-622 for verifying the actual output events 641-642 that are receivedfrom the event processing system 620. The verification process caninvolve different criterions. For example, the testing framework 600 canensure that the expected output events 621-622 and actual output events641-642 are the same kind and have matching timestamps (i.e. thetimestamp 631 matches the timestamp 651 and the timestamp 632 matchesthe timestamp 652) and other properties.

Furthermore, the testing framework 600 can apply a tolerance on theverification process. The tolerance can be used for accommodating thelatencies in various tests, since the event processing system 600 maynot generate the output event exactly at the expected time, e.g. due tothe time consumed by the event processing system 600 for performing therule execution and event delivery.

FIG. 7 shows an illustration of verifying the outputs of a query fortesting an event processing system, in accordance with an embodiment ofthe invention. As shown in FIG. 7, at step 701, the testing framework700 can start the verification process.

At steps 702-703, the testing framework 700 can obtain the current time(e.g. a system time t0), and checks whether there are available inputevents.

If there are available input events, at step 704, the testing framework700 can obtain the next input event. Furthermore, at steps 705-707, thetesting framework 700 can obtain the timestamp (t) for the input event,and waits until the time (t0+t). Then, the testing framework 700 cansend the input event to the event processing system before going back tothe step 703 for checking whether there are more available input events.

On the other hand, if there are no more input events, at step 708, thetesting framework 700 can check whether there are available expected(output) events. At step 716, the verification process completes whenthere are no more expected events.

Otherwise, when there are more expected events, at step 709, the testingframework 700 can obtain the next expected event. Then, at steps710-711, the testing framework 700 can obtain the next actual event andobtains the timestamp (t′) for the expected event.

Moreover, at step 712, the testing framework 700 can check whether thetimestamp of the actual event matches the expected time (i.e. t0+t′). Atstep 713, the testing framework 700 can check whether the kind of theexpected event matches the actual event. At step 714, the testingframework 700 can check whether the different properties of the expectedevent matches the actual event.

At step 715, the testing framework 700 may determine that theverification of the assert statement fails if the checks in one or moreof the above steps 712-174 fails. Otherwise, the testing framework 700can go back to the step 708 checking whether there are more expectedevents and proceeds accordingly.

FIG. 8 shows an illustration of testing an event processing system basedon a system time, in accordance with an embodiment of the invention. Asshown in FIG. 8, when the test 800 is executed, the testing frameworkcan send one or more input events 810, which are based on a system time830, to an event processing system. For example, the input events 810includes the events e1-e6 with the system timestamps t1-t6.

In accordance with an embodiment of the invention, the testing framework800 can move an event window 820 along with the system time 830. Asshown in FIG. 8, the testing framework can move the event window 820from the window 801 to the window 806 as the events e1-e6 are receivedby the event processing system.

At the time t1, an event e1 is inserted into the event stream that isreceived by the event processing system. The system can add the event e1into the event window 801 since the event e1 is a new event. Also, theresult set 840 indicates that the event processing system is expected tooutput an insert event 841 for the new event e1.

Furthermore, the testing framework can add more new events, such as e2and e3, in the windows 802-803 respectively, when the new events areinserted into the stream. The result set 840 indicates that the eventprocessing system is expected to output an insert event 841 for each ofthe new event e2 and e3 separately.

At the time t4, the system can add an event e4, when it is inserted intothe event stream. At the same time, the event e1 may be deleted from thewindow 804, if the event e1 becomes an old event or obsolete (e.g., whenthe time separation (t4−t1) is longer than the event window 820). Thus,the event processing system is expected to output an insert event 841for the new event e1 and a delete event 842 for the deleted event e1.

At the time t5, a new event e5 is inserted into the event stream (i.e.in the window 805). The result set 840 indicates that the eventprocessing system is expected to output an insert event 841 for the newevent e5.

At the time t6, a new event e6 is inserted into the stream (i.e. thewindow 806). The result set 840 indicates that the event processingsystem is expected to output an insert event 841 for the new event e6and a delete event 842 for the deleted event e2.

The following List 2 is an exemplary XML test file for testing an eventprocessing system using an event window that is based on a system time.

List 2 <input-channel id=″s1″ event-type=″StockEvent″/> <output-channelid=″s3″ event-type=″StockEvent″/> <rule id=”rule1” type=”query”>SELECT * FROM s1[RANGE 3] </rule> <input-events channel=”s1”><StockEvent symbol=”ABC” price=”19.8” createTime=”10080900”timestamp=”0” kind=”+”/> <StockEvent symbol=”ABC” price=”19.9”createTime=”10080901” timestamp=”1000” kind=”+”/> <Wait time=”2000”/></input-events> <output-events channel=”s3” verify=”all”> <StockEventsymbol=”ABC” price=”19.8” createTime=” 10080900” timestamp=”0”kind=”+”/> <StockEvent symbol=”ABC” price=”19.9” createTime=” 10080901”timestamp=”1000” kind=”+”/> <StockEvent symbol=”ABC” price=”19.8”createTime=”10080900” timestamp=”3000” kind=”−”/> </output-events>

As shown in the above List 2, the testing framework includes an eventchannel s1 for inputting StockEvents and an event channel s3 foroutputting StockEvents. Furthermore, the event window, as defined in therule section, is specified as [RANGE 3 SECONDS], which indicates thatthe query computes the events from the current time back three (3)seconds.

When the test is executed, a first event, e.g. <StockEvent symbol=“ABC”price=“19.8” createTime=“10080900” timestamp=“0”/>, is inserted. Thetesting framework can use a system time t1 to represent the time whenthe first event is inserted, since the system time represents onlyrelative time. Thus, the system time t1 can serve as the base time (or abaseline) for the stream of events. For example, the value of the basetime t1 may be set as 0, even though the real value of the time is adifferent number (e.g. 1980911331990) in the system.

Furthermore, the events can be outputted immediately after the query isexecuted. In the above test, the expected output event at the time t1can be specified as <StockEvent symbol=“ABC” price=“19.8”createTime=“10080900” timestamp=“0” kind=“+”/>, which indicates that aninsert event is expected to be outputted by the event processing systemat the time when the event e1 is inserted.

Additionally, the testing framework receives an actual output event fromthe event processing system. The testing framework can check whether thevalue of the pseudo column “time” for the actual event is equal to thesum of the base time, t1, plus the value of the timestamp of theexpected event.

For example, the timestamp for the second event, <StockEventsymbol=“ABC” price=“19.9” createTime=“10080901” timestamp=“1000”/>, isspecified as one thousand (1000) milliseconds. Thus, the second event isexpected to be generated by the event processing system one (1) secondafter the first event.

Additionally, the test file includes a special input, <Waittime=“2000”/>, which indicates that the input may be blocked for two (2)seconds. Thus, the end time for the special input is three (3) secondsafter the first event (i.e. t1+3000), at which time the system isexpected to generate a delete event, <StockEvent symbol=“ABC”price=“19.8” createTime=“10080900” timestamp=“3000” kind=“−”/>.

Then, the testing framework can compare the actual time of the deleteevent with the time, at which the first event is inserted (t1), andchecks if the difference is three thousand (3000) milliseconds (e.g.with a tolerance applied).

FIG. 9 shows an illustration of testing an event processing system basedon an application time, in accordance with an embodiment of theinvention. As shown in FIG. 9, when a test 900 is executed, the testingframework sends one or more input events 910 (e.g. events e1-e5) to anevent processing system along the application time 930 (whichcorresponds to a system time 950).

At the application time ts1 (which corresponds to a system time t1), anevent e1 is inserted into the event stream, and ts1 is attached to theevent e1 as an application timestamp. Since the event e1 is a new event,the system can add the event e1 into the event window 901. Thus, theevent processing system is expected to output an insert event 941 forthe new event e1, as indicated in the result set 940.

Furthermore, at the application time ts2 and ts3 (which may bothcorrespond to a system time t1), the new events e2 and e3 can beinserted into the stream (the windows 902-903) respectively. The resultset 940 indicates that the event processing system is expected to outputan insert event 941 for each of the new event e2 and e3 separately.

As shown in FIG. 9, at the application time of ts4 (which corresponds toa system time t3), a heartbeat event can be inserted into the eventstream, which moves the event window 920 to the window 904. Also at thistime, the event e1 may become an old event, e.g., when the length of thetime separation (ts4−ts1) is greater than the length of the event window920, and the system may delete the event e1 from the window 904. Thus,the event processing system is expected to output a delete event 942 forthe deleted event e1.

Additionally, at the application time ts5 (which corresponds to a systemtime t4), a new event e4 is inserted into the stream (the window 905).The result set 940 indicates that the event processing system isexpected to output an insert event 941 for the new event e4.

At the application time ts6, which may also correspond to a system timet4, a new event e5 is inserted into the stream (the window 906). Theresult set 940 indicates that the event processing system should outputan insert event 941 for the new event e5 and a delete event 942 for thedeleted event e2.

The following List 3 is an exemplary XML test file for testing an eventprocessing system using an event window based on an application time.

List 3 <input-channel id=″s1″ event-type=″StockEvent″><application-timestamped> <expression>createTime</expression></application-timestamped> </input-channel> <output-channel id=″s3″event-type=″StockEvent″/> <rule id=”rule1” type=”query”> SELECT * FROMs1[RANGE 3] </rule> <input-events channel=”s1”> <StockEvent symbol=”ABC”price=”19.8” createTime=”1000” kind=”+”/> <HeartBeat timestamp=”4000”/></input-events> <output-events channel=”s3” verify=”all”> <StockEventsymbol=”ABC” price=”19.8” createTime=”1000” timestamp=”1000” kind=”+”/><StockEvent symbol=”ABC” price=”19.8” createTime=”1000” timestamp=”3000”kind=”−”/> </output-events>

As shown in the above List 3, the testing framework includes an eventchannel s1 for inputting StockEvents and an event channel s3 foroutputting StockEvents.

Additionally, the event channel s1 is configured with an applicationtimestamp expression, which indicates that the result of the expressioncreateTime is used as the timestamp of the event (in such a case, theexpression result is the value of the attribute createTime of theevent).

Furthermore, the event window, as defined in the rule section, is [RANGE3], which indicates that the query computes the events from the currenttime back three (3) seconds. Also, the processor is informed that theevent window in the query, SELECT * FROM s1[RANGE 3], is subject to thevalue of the expression createTime.

When the test is executed, an event, <StockEvent symbol=“ABC”price=“19.8” createTime=“1000” kind=“+”/>, is inserted into the streams1. For an application timestamped stream, the time for an input eventis either the value of timestamp property of the event or the value ofheartbeat. In the rule engine, the associated timestamp of the event isin the application time (e.g. 1000 milliseconds), which is the value ofthe createTime attribute.

On the other hand, for an output event, the value of the time propertyis defined in the “time” pseudo column of the actual event and in thetimestamp property of the expected event (the value of the createTimeattribute is not used as the timestamp, even though the value may be thesame with the timestamp).

According to the rule, the expected output event, <StockEventsymbol=“ABC” price=“19.8” createTime=“1000” timestamp=“1000” kind=“+”/>,is an insert event at the time 1000 milliseconds, which is an absolutevalue.

When the testing framework receives the actual output event, the testingframework constructs an internal representation, by setting the property“timestamp” to be the value of the pseudo column “time” of the actualevent and setting the property “kind” according to the pseudo column“kind” of the actual event.

For verification purpose, the testing framework checks whether theactual output event is an insert event, as indicated by the value “+”for the kind property in the expected output event. Also, the testingframework checks whether the value of the pseudo column “time” of theactual event, is equal exactly to the value of the property “timestamp”of the expected output event.

As shown in the above List 3, a heartbeat, e.g. <HeartBeattimestamp=“4000”/>, can be sent to the rule engine, which indicates thatthe current application time is 4000 milliseconds. Also, the<HeartBeat/> element may be a special event (input) recognizable by theframework, which is not defined in the schema.

In accordance with an embodiment of the invention, using a heartbeat,the time moves a unit by a unit while appearing elapsed normally. Thus,the testing framework expects a delete event, <StockEvent symbol=“ABC”price=“19.8” createTime=“1000” timestamp=“3000” kind=“−”/>, since theoutput event is triggered at the time 4000 milliseconds (greater than 3seconds).

Then, the testing framework checks whether the actual event is a deleteevent, and checks whether the value of pseudo column “time” is equal to3000 milliseconds, as indicated by the value of the timestamp propertyof the expected event.

FIG. 10 illustrates an exemplary flow chart for using an event windowfor testing an event processing system, in accordance with an embodimentof the invention. As shown in FIG. 10, at step 1001, a testing frameworkoperates to send one or more input events in an event stream to an eventprocessing system, wherein each said input event is associated with atimestamp. Furthermore, at step 1002, the testing framework can receiveone or more actual output events from the event processing system,wherein each said actual output event is associated with a timestamp.Then, at step 1003, the testing framework can compare said one or moreactual output events in an event window with one or more expected outputevents for the event window.

Sliding Window Using an Application Time

FIG. 11 shows an illustration of supporting a sliding window for testingan event processing system based on an application time, in accordancewith an embodiment of the invention. As shown in FIG. 11, a testingframework 1100 can provide an event stream 1160 to an event processingsystem 1120, which processes various events in a batch mode. Each of theinput events 1101-1103 can be associated with a timestamp, such as thetimestamps 1111-1113 that are based on an application time.

In accordance with an embodiment of the invention, the testing framework1100 can apply an event window, such as a sliding window 1110, on theevent stream 1160. The sliding window 1110 can include one or more inputevents 1101-1103, which are received and processed by the eventprocessing system 1120 within a sliding cycle 1140 (i.e. in a batch).

Furthermore, the testing framework 1100 can receive one or more actualoutput events 1150, such as the output events 1141-1142, from the eventprocessing system based on the rule processing. Each of the actualoutput events 1141-1142 can be associated with an application time stamp1151-1152.

In accordance with an embodiment of the invention, the testing framework1100 can prescribe one or more expected output events 1130, such as theoutput events 1121-1122, which share a time stamp 1131. These expectedoutput events 1121-1122 can be used for verifying the actual outputevents 1141-1142, which are received from the event processing system1120.

In accordance with an embodiment of the invention, the testing framework1100 is able to verify the output events, which are generated by theevent processing system 1120 in a random order. For example, the testingframework 1100 can verify whether the application timestamps 1151-1152associated with said actual output events 1141-1142 matches the sharedapplication timestamp 1131 associated with said one or more expectedoutput events 1121-1122 for the sliding window 1110 (e.g. with atolerance 1161).

As shown in FIG. 11, the output events 1141 and 1142, which are expectedto be generated at the same time, may actually be generated at differenttimes (i.e., with different timestamps 1151 and 1152). The order for theoutput events 1141 and 1142 can be random and may be subject to theexact time that each event is actually generated. Thus, the testingframework 1100 may not be able to compare the actual output events andthe expected output events in a sequential order (i.e. one by one).

FIG. 12 shows an illustration of verifying the outputs of the query witha system timestamp for testing an event processing system, in accordancewith an embodiment of the invention. As shown in FIG. 12, at step 1201,the testing framework 1200 can start the verification process.

Then, at step 1202, the testing framework 1200 can check whether thereare available expected output events. If there are no expected events,at step 1212, the testing framework 1200 can end the verificationprocess (and confirm that the assert verification is successful).

Otherwise, at steps 1203-1204, the testing framework 1200 can obtain thenext expected output event if it is available and obtain the next actualoutput event.

At step 1205, the testing framework 1200 can check whether the actualoutput event matches the expected output event. If the actual outputevent matches the expected output event, the testing framework 1200 cancheck whether there are more expected output events at step 1202 andthen proceeds accordingly.

At steps 1206-1207, if the actual output event does not match theexpected output event, the testing framework 1200 can obtain the nextexpected output event, and check whether there are two or more expectedevents have the same timestamp. At steps 1211-1212, if there are noexpected events that share the same timestamp, the testing framework1200 may conclude that the assert fails before ending the verificationprocess.

On the other hand, there may be at least two expected events that sharethe same timestamp. At step 1208, the testing framework 1200 can obtainall expected output events that share the same timestamp. Also, at step1209, the testing framework 1200 can obtain all actual output eventsthat have the same timestamp as the current actual event.

At step 1210, the testing framework 1200 can check whether the events inthe two sets match. If the events in the two sets match, at step 1202,the testing framework 1200 can check whether there are more expectedoutput events. If the events in the two sets do not match, at step 1211,the testing framework 1200 may conclude that the assert fails beforeending the verification process.

FIG. 13 shows an illustration of testing an event processing systemusing a sliding window based on an application time, in accordance withan embodiment of the invention. As shown in FIG. 13, when a test 1300 isexecuted, the testing framework can insert one or more input events 1310(e.g. events e1-e6 with application timestamps ts1-ts6) to an eventprocessing system.

The testing framework 1300 can move the sliding window 1320 from thewindow 1301 to the window 1302 along the application time 1330, as theevents e1-e6 are received and processed in a batch mode by an eventprocessing system.

As indicated in the result set 1340, the testing framework 1300 expectsthe output events to be generated at the end of each sliding window1301-1302. For example, at the end of the window 1301, the eventprocessing system can generate an insert event 1341 for the eventse1-e3. At the end of the window 1302, the event processing system cangenerate an insert event 1341 for the events e4-e6, and a delete event1342 for the old events e1-e3.

The following List 4 is an exemplary XML test file for using a slidingwindow for testing an event processing system based on an applicationtime.

List 4 <input-channel id=″s1″ event-type=″StockEvent″><application-timestamped> <expression> createTime </expression></application-timestamped> </input-channel> <output-channel id=″s3″event-type=″StockEvent″/> <rule id=”rule1” type=”query”> SELECT *FROM s1[RANGE 3 SLIDE 3] </rule> <input-events channel=”s1”> <StockEventsymbol=”ABC” price=”19.8” createTime =”0” > <StockEvent symbol=”ABC”price=”19.9” createTime =”1000”/> <StockEvent symbol=”ABC” price=”20.1”createTime =”3001”/> <StockEvent symbol=”ABC” price=”21.2” createTime=”6000”/> </input-events> <output-events channel=”s3”> <StockEventsymbol=”ABC” price=”19.9” createTime=”0” <timestamp=”3000” kind=”+”/><StockEvent symbol=”ABC” price=”19.8” createTime=”1000” timestamp=”3000”kind=”+”/> <StockEvent symbol=”ABC” price=”20.1” createTime=”3001”timestamp=”6000” kind=”+”/> <StockEvent symbol=”ABC” price=”19.8”createTime=”0” timestamp=”6000” kind=”−”/> <StockEvent symbol=”ABC”price=”21.2” createTime=”6000” timestamp=”6000” kind=”+”/> StockEventsymbol=”ABC” price=”19.9” createTime=”1000” timestamp=”6000” kind=”−”/></output-events>

As shown in the above, the testing framework includes an input channels1 for inputting StockEvents and an output channel s3 for outputtingStockEvents. Additionally, the event channel s1 is configured with anapplication timestamp expression, which indicates that the result of theexpression createTime is the timestamp of the event.

Furthermore, the processor is informed that the event window in thequery is subject to the value of createTime. Also, the query, SELECT *FROM s1[RANGE 3 SLIDE 3], which indicates that, for every three (3)seconds, the rule engine computes the events in an event window with alength of three (3) seconds, and the output events are generated everythree (3) seconds.

As shown in the above List 4, the sliding window moves, when the event,<StockEvent symbol=“ABC” price=“20.1” createTime=“3001”/>, is inserted.Since the event processing system only generates output events at theend of each sliding windows, the system may generate output for theinput events, <StockEvent symbol=“ABC” price=“19.8” createTime=“0”> and<StockEvent symbol=“ABC” price=“19.9” createTime=“1000”/>, at the timeof 3000 milliseconds.

The output events of the rule processor are expected to be shown in thefollowing.

<StockEvent symbol=”ABC” price=”19.9” createTime=”0” timestamp=”3000”kind=”+”/> <StockEvent symbol=”ABC” price=”19.8” createTime=”1000”timestamp=”3000” kind=”+”/>

Both the first event and the second event are associated with thetimestamp “3000”, which indicates that the two events are outputted atthe same time. Also, it is possible that the events are received indifferent orders. In other words, the above result is equivalent to thefollowing.

<StockEvent symbol=”ABC” price=”19.8” createTime=”1000” timestamp=”3000”kind=”+”/> <StockEvent symbol=”ABC” price=”19.9” createTime=”0”timestamp=”3000” kind=”+”/>

Subsequently, the sliding window moves ahead as the event, <StockEventsymbol=“ABC” price=“21.2” timestamp=“6000”/>, is inserted into the eventstream. At such a time, two new events are added into the window, twoold events are deleted from the window. Thus, the output of the querycan be shown as in the following.

<StockEvent symbol=”ABC” price=”19.9” createTime=”1000” timestamp=”6000”kind=”−”/> <StockEvent symbol=”ABC” price=”19.8” createTime=”0”timestamp=”6000” kind=”−”/> <StockEvent symbol=”ABC” price=”20.1”createTime=”3001” timestamp=”6000” kind=”+”/> <StockEvent symbol=”ABC”price=”21.2” createTime=”6000” timestamp=”6000” kind=”+”/>

In accordance with an embodiment of the invention, users are allowed tospecify the events in different orders in the test file. In the aboveexample, the test is considered successful even when the order of theoutput events does not match with the order of the expected events,since all the events are outputted at the time 6000 milliseconds.

FIG. 14 illustrates an exemplary flow chart for supporting a slidingwindow for testing an event processing system based on an applicationtime, in accordance with an embodiment of the invention. As shown inFIG. 14, at step 1401, a testing framework can apply a sliding window ona stream of input events that are sent to an event processing system,wherein said event processing system processes the stream of inputevents in a batch mode with one or more sliding cycles, and wherein saidsliding window includes one or more input events in a sliding cycle thatcorrespond to a plurality of expected output events. Then, at step 1402,the testing framework operates to receive a plurality of actual outputevents at an end of said sliding cycle in the batch mode from the eventprocessing system. Furthermore, at step 1403, the testing framework canuse said one or more expected output events to verify said one or moreactual output events.

Sliding Window Using a System Time

FIG. 15 shows an illustration of supporting a sliding window for testingan event processing system based on a system time, in accordance with anembodiment of the invention. As shown in FIG. 15, a testing framework1500 can provide an event stream 1560 to an event processing system 1520based on a system time 1570. The event stream 1560 includes a pluralityof input events 1501-1504.

Furthermore, the event processing system 1520 can process the events1501-1504 in the event stream 1560 in a batch mode. For example, theevent processing system 1520 can process the events 1502-1504, which arein a sliding cycle 1540, in a batch.

As shown in FIG. 15, the input events 1501-1504 can be associated withthe timestamps 1511-1514, which are based on a system time 1570. Thevalue of the timestamp (e.g. the pseudo timestamp property) based on thesystem time 1570 is a relative value, which indicates the timeseparation between the time when an event is inserted and the initialtime when the first event was inserted.

In accordance with an embodiment of the invention, the testing framework1500 can apply an event window, e.g. a sliding window 1510, on the eventstream 1560. The sliding window 1510 may need to match the slidingwindow 1510 with the sliding cycle 1540, since the value of thetimestamp property specified in the system time 1570 may not be used asthe real timestamp of the event (unlike an event stream associated withan application timestamp).

For example, the testing framework 1500 can obtain the sliding point1571 in the system time 1570 for the sliding cycle 1540. The testingframework 1500 can use the sliding point 1571 as a base time forcalculating the output events 1530 based on the value of the configuredtimestamp 1512-1514 of the input events 1502-1504.

As shown in FIG. 15, the sliding window 1510 can include one or moreinput events 1512-1514, which are received and processed in the slidingcycle 1540 by the event processing system 1520.

Thus, the testing framework 1500 can correctly match the sliding window1510 in the system time 1570 with the input events 1502-1504 that areprocessed in a batch mode by the event processing system 1520. Also, thetesting framework 1500 can prevent the query associated with the slidingwindow 1510 from generating the output events 1550 arbitrarily.

In accordance with an embodiment of the invention, the testing framework1500 can prescribe one or more expected output events 1530, whichincludes the output events 1521-1522 that share a time stamp 1531.

Furthermore, the expected output events 1521-1522 can be used forverifying the actual output events 1550 that are received from the eventprocessing system 1520. For example, the testing framework 1500 canverify whether the system timestamp 1551-1552, which is associated witheach actual output event 1541-1542 received from the event processingsystem, matches the shared system timestamp 1531 (with tolerance 1561),which is associated with said one or more expected output events1521-1522 in the sliding window 1510.

Additionally, a tolerance 1561 can be applied to accommodate the latencyin the process.

FIG. 16 shows an illustration of testing an event processing systemusing a sliding window based on a system time, in accordance with anembodiment of the invention. As shown in FIG. 16, when a testingframework execute the test 1600, the sliding window 1620 movesaccordingly to the system time 1630.

For example, the sliding window 1620 may start at an arbitrary systemtime, to, from which time the windows 1601-1603 can be calculated (sincethe query may be started at any time). Furthermore, the testingframework can send one or more input events 1610 (e.g. the events e1-e4with the system timestamps t1-t4) to the event processing system.

As shown in FIG. 16, at the time t1 (e.g. with a value 0), the firstevent e1 can be inserted with a system timestamp t1. The separationbetween the time t1 and the starting time, t0, of the sliding window1620 can be arbitrary.

Thus, the testing framework may need to correctly determine a base timefor the windows 1602-1603. Otherwise, the testing framework 1600 may notbe able to ensure that the event windows 1602-1603 match the slidingcycles correctly, since the output events can be random.

As shown in FIG. 16, using a batch mode, the event processing system cangenerate an output event (an insert event 1641 for the event e1) at theend of the window 1601 (i.e. the sliding point 1631) in the result set1640. The testing framework 1600 can use the sliding point 1631 as thebase time (or baseline) for determining the subsequent sliding windows1602-1603, which matches the sliding cycles. Thus, the testing framework1600 is able to determining the output events correctly.

As shown in FIG. 16, an input event, e2, can be inserted into the eventstream at the time t2, and an input event, e3, can be inserted into theevent stream at the time t3. Thus, at the end of the window 1602, theevent processing system may generate an insert event 1641 for the evente2-e3 and a delete event 1642 for the event e1 (since the first event e1may become obsolete).

Additionally, at the time t4, an input event, e4, can be inserted in tothe event stream. Thus, at the end of the window 1603, the eventprocessing system may generate an insert event 1641 for the event e4,and a delete event 1642 for the event e2-e3 (since the events e2-e3 maybecome obsolete).

The following List 6 is an exemplary XML test file for testing an eventprocessing system using a sliding window based on a system time.

List 6 <input-channel id=″s1″ event-type=″StockEvent″/> <output-channelid=″s3″ event-type=″StockEvent″/> <rule id=”rule1” type=”query”>SELECT * FROM s1[RANGE 3 SLIDE 3] </rule> <assert> <input-eventschannel=”s1”> <StockEvent symbol=”ABC” price=”19.8” timestamp=”1000” ><input-events> <output-events channel=”s3” verify=”any”/> </assert><assert> <input-events channel=”s1”> <StockEvent symbol=”ABC”price=”19.9” timestamp =”1000”/> <StockEvent symbol=”ABC” price=”20.1”timestamp =”2000”/> <StockEvent symbol=”ABC” price=”21.2” timestamp=”6000”/> </input-events> <output-events channel=”s3”> <StockEventsymbol=”ABC” price=”19.9” timestamp=”0” kind=”−”/> <StockEventsymbol=”ABC” price=”19.8” timestamp=”3000” kind=”+”/> <StockEventsymbol=”ABC” price=”20.1” timestamp=”3000” kind=”+”/> <StockEventsymbol=”ABC” price=”19.8” timestamp=”6000” kind=”−”/> <StockEventsymbol=”ABC” price=”21.2” timestamp=”6000” kind=”+”/> <StockEventsymbol=”ABC” price=”19.9” timestamp=”6000” kind=”−”/> </output-events></assert>

As shown in the above, the testing framework includes an input channels1 for inputting StockEvents and an output channel s3 for outputtingStockEvents.

Furthermore, the event window defines a query, SELECT * FROM s1[RANGE 3SLIDE 3], which indicates that the rule engine can periodically computethe events within a range of three (3) seconds (for every three (3)seconds). As a result, the output events can be generated at the end ofevery three (3) seconds.

The testing framework can take advantage of a configuration,<output-events channel=“s3” verify=“any”/>, which is defined in thefirst assert in the above. The value any for the attribute verifyindicates that any insert event may satisfy the verify condition.

Thus, the testing framework may generate the first output event at theend of the first sliding window (e.g. t0+9000 milliseconds) for theevent <StockEvent symbol=“ABC” price=“19.8” timestamp=“1000”>. Then, thetesting framework can determine the sliding point accordingly in orderto verify the outputs.

Once the processing of the first assert is complete, the testingframework may process the next assert, which starts exactly (orapproximately) at the time of the sliding point (i.e. the sliding pointis used as a base time). As a result, the input event <StockEventsymbol=“ABC” price=“19.9” timestamp=“1000”/> may be inserted at thetime, t0+10000 milliseconds, the event <StockEvent symbol=“ABC”price=“20.1” timestamp=“2000”/> may be inserted at the time, t0+11000milliseconds, and the event <StockEvent symbol=“ABC” price=“21.2”timestamp=“6000”/> may be inserted at the time, t0+15000 milliseconds.

Also, the testing framework can generate the output events accordingly.For example, the event, <StockEvent symbol=“ABC” price=“19.9”timestamp=“0” kind=“−”/>, the event, <StockEvent symbol=“ABC”price=“19.8” timestamp=“3000” kind=“+”/>, and the event, <StockEventsymbol=“ABC” price=“20.1” timestamp=“3000” kind=“+”/> may be outputtedat the time, t0+12000 milliseconds.

Additionally, the event, <StockEvent symbol=“ABC” price=“19.8”timestamp=“6000” kind=“−”/>, =“−”/>, the event, <StockEvent symbol=“ABC”price=“21.2” timestamp=“6000” kind=“+”/>, and the event, <StockEventsymbol=“ABC” price=“19.9” timestamp=“6000” kind=“−”/> may be outputtedat the time, t0+15000 milliseconds.

Thus, the testing framework can verify the output of a query, which isbased on a sliding window that is applied on the system timestampedstream.

FIG. 17 illustrates an exemplary flow chart for supporting a slidingwindow for testing an event processing system based on a system time, inaccordance with an embodiment of the invention. As shown in FIG. 17, atstep 1701, a testing framework operates to send a stream of input eventsfor an event processing system, wherein each said input event isassociated with a timestamp that is based on a system time, and whereinsaid event processing system processes the stream of input events in abatch mode with one or more sliding cycles. Furthermore, at step 1702,the testing framework can determine a base time (or a baseline) for anevent window in the system time, wherein said event window includes oneor more input events in a sliding cycle that corresponds to a pluralityof expected output events associated with a shared timestamp. Then, atstep 1703, the testing framework can apply the event window on thestream of input events that are sent to the event processing system.

Multiple Input Event Streams

FIG. 18 shows an illustration of supporting testing an event processingsystem with multiple input event streams, in accordance with anembodiment of the invention. As shown in FIG. 18, a testing framework1800 can provide multiple event streams 1881-1882 to an event processingsystem 1820 in parallel (e.g. using different channels).

In accordance with an embodiment of the invention, the testing framework1800 can avoid processing the input events sequentially (e.g., in theorder that they are presented in the test file). For example, each eventstream 1881-1882 can be configured with a separate connection, aseparate event channel, and a separate thread. Also, the testingframework 1800 can insert multiple events into the different eventstreams 1881-1882 at the same time.

Furthermore, the testing framework 1800 can apply separate event windowson different event streams 1881-1882. As shown in FIG. 18, an eventwindow 1810, which includes input events 1801-1803, is applied on theevent stream 1881; and an event window 1830, which includes input events1821-1822, is applied on the event stream 1882.

Additionally, each of the input events can be associated with atimestamp based on either an application time or a system time. Forexample, the input events 1801-1803 can be associated with a timestamp1811-1813, and the input events 1821-1822 can be associated with atimestamp 1831-1832.

In accordance with an embodiment of the invention, the events indifferent streams 1881-1882 (or event channels) can share the same basetime (or baseline). The base time for a sliding window in an applicationtimestamped event stream can be the time when the framework starts toprocess the first assert in the test. On the other hand, the base timefor a sliding window in a system timestamped event stream can be thetime when the framework starts to process the first assert afterobtaining a corresponding sliding point (e.g. the second assert).

Additionally, a query 1840 can be provided to the event processingsystem 1820 for relating multiple parallel event streams 1881-1882 inevent processing. For example, the testing framework 1800 can receiveone or more actual output events 1851 from the event processing system1820 based on the rule processing. Furthermore, the testing framework1800 can correlate the various input events 1801-1803 and 1821-1822 indifferent event streams 1881-1882 to generate one or more expectedoutput events 1841.

Then, the testing framework 1800 can verify the actual output events1851 using the expected output events 1841. For example, the testingframework 1800 can use the expected output events 1841 with a time stamp1861 to verify the actual output events 1851 with a time stamp 1871.

FIG. 19 shows an illustration of using multiple event windows fortesting an event processing system, in accordance with an embodiment ofthe invention. As shown in FIG. 19, when the test 1900 is executed, thetesting framework can send a plurality of input events 1910 in multiplestreams to an event processing system as the time 1930 moves from t1 tot10. For example, the events e1-e6 are sent via the channel 1911 and theevents w1-w5 are sent via the channel 1912.

Additionally, the event windows 1920 that are applied on the differentevent channels can be configured differently (i.e., the testingframework can apply different event windows on the event channelsseparately). For example, the event windows 1901, 1907 and 1909 windows,which are applied on the event channels 1911, can be configured as timebased, while the event windows 1921, 1927 and 1929, which are applied onthe event channels 1912, can be configured as event based.

As shown in FIG. 19, when the test 1900 is executed, an event e1 isinserted into the event window 1901 via the channel 1911, at the timet1. Then, at the time t2, another event w1 can be inserted via thechannel 1912. Based on the rule processing, at the time t2, the eventprocessing system is expected to generate an output event p1 for theinserted events e1 and w1 in the result set 1940.

Furthermore, an event e2 is inserted in the channel 1911 at the time t3and an event w2 is inserted in the channel 1912 at the time t4. Based onthe rule processing, the event processing system is expected to generatean output event p2 for the inserted events e2 and w2, at the time t4.

At the time t5, an event e3 is inserted in the channel 1911 and an eventw2 is inserted in the channel 1912. Based on the rule processing, theevent processing system is expected to generate an output event p3 forthe inserted events e3 and w3, at the time t5.

Additionally, an event e4 is inserted in the channel 1911 at the timet6, and an event w4 is inserted in the channel 1912 at the time t7.Based on the rule processing, the event processing system is expected togenerate an output event p4 for the inserted events e2 and w2, at thetime t7.

Moreover, an event w5 is inserted in the channel 1912 at the time t8 andthe event e5 is inserted in the channel 1912 at the time t9. Based onthe rule processing, an output event p5 is expected based on theinserted events e5 and w5. Also, an event e6 is inserted in the channel1912 at the time t10, at the time t9.

The following List 7 is an exemplary XML test file for testing an eventprocessing system with multiple input event streams.

List 7 <input-channel id=″s1″ event-type=″StockEvent″/> <input-channelid=″s2″ event-type=″OrderEvent″/> <output-channel id=″s3″event-type=″TradeEvent″/> <rule id=”rule1” type=”query”> SELECTs2.product as product, s1.price as price, s1.createTime as tradeTimeFROM s1[RANGE 4] , s2[ROWS 2] WHERE s1.symbol=s2.symbol </rule><input-events channel=”s1” parallel=”true”> <StockEvent symbol=”ABC”price=”19.8” createTime=”10080900” timestamp=”0” /> <StockEventsymbol=”AAA” price=”19.9” createTime=”10080901” timestamp=”2000” /></input-events> <input-events channel=”s2” parallel=”true”> <OrderEventsymbol=”ABC” product=”STOCK” customer=”D1” timestamp=”1000” /><OrderEvent symbol=”AAA” product=”SECURITIES” customer=”D2”timestamp=”2000” /> </input-events> <output-events channel=”s3”verify=”all”> <TradeEvent product=”STOCK” price=”19.8”tradeTime=”10080900” timestamp=”1000” kind=”+”/> <TradeEventproduct=”SECURITIES” price=”19.9” tradeTime=”10080901” timestamp=”2000”kind=”+”/> </output-events>

As shown in the above, the testing framework includes an event channels1 for inputting StockEvents and an event channel s2 for inputtingOrderEvents. Additionally, the testing framework includes an eventchannel s3 for outputting StockEvents.

Furthermore, as defined in the rule section, the event window applied onthe input channel, [RANGE 3 SECONDS], indicates that the query computesthe events from the current time back three (3) seconds. Also, the eventwindow on the input channel s2, [Rows 2], indicates that the querycomputes the events for the last two rows.

As shown in the above List 7, the testing framework can include a“parallel” attribute in the <assert> element. The “parallel” attributeindicates that the testing framework may process the input events inseparate threads and all the events in different channels can share thesame base time (e.g., the time when the testing framework starts toprocess the first <assert> statement of the test).

For example, when the testing framework detects that “parallel”attribute for the event channel s1 is set to be “true”, the testingframework can proceed to check the “parallel” attributes for theremaining <input-event/> elements. Also, if the value for the “parallel”attribute is “true”, the framework can allocate multiple threads andassigns the jobs to process the <input-events/> using different threads.Thus, the testing framework can avoid process the input-eventssequentially, such as in the order that are presented in the test file.

When the test is executed, the testing framework can insert a stockevent, <StockEvent symbol=“ABC” price=“19.8” createTime=“10080900”timestamp=“0”/>, to the channel s1 at time 0, and can insert an orderevent <OrderEvent symbol=“ABC” product=“STOCK” customer=“D1”timestamp=“1000”/> to the channel s2 at time 1000.

Additionally, the testing framework can insert a stock event <StockEventsymbol=“AAA” price=“19.9” createTime=“10080901” timestamp=“2000”/> tothe channel s1, and insert an order event <OrderEvent symbol=“AAA”product=“SECURITIES” customer=“D2” timestamp=“2000”/> to the channel s2,at the same time of 2000.

Thus, the testing framework can use a first thread to process the eventsin stream s1 with a starting time t1. Therefore, the real time for thefirst event in stream s1 is t1+0, and the real time for the second eventin stream s1 is t1+2000. Also, the testing framework can use a secondthread to process the events in stream s2, with the starting time t1.The real time for the first event in stream s2 is t1+1000, and the realtime for the second event in stream s2 is t1+2000.

As shown in the above List 7, the user can specify the order for theevents. Also, the order of the last two events may be specifieddifferent, since the event processing system may not be able to insertthe last two events exactly at the same time (i.e. t1+2000).

FIG. 20 illustrates an exemplary flow chart for supporting testing anevent processing system with multiple input event streams, in accordancewith an embodiment of the invention. As shown in FIG. 20, at step 2001,a testing framework operates to send a plurality of input events inmultiple event streams to an event processing system, wherein each saidinput event is associated with a timestamp. Furthermore, at step 2002,the testing framework can prescribe one or more expected output eventsthat correspond to said plurality of input events in multiple eventstreams, wherein each said expected output event is associated with atimestamp. Then, at step 2003, the testing framework can use said one ormore expected output events to verify one or more actual output eventsthat are received from the event processing system, wherein each saidactual output event is associated with a timestamp.

The present invention may be conveniently implemented using one or moreconventional general purpose or specialized digital computer, computingdevice, machine, or microprocessor, including one or more processors,memory and/or computer readable storage media programmed according tothe teachings of the present disclosure. Appropriate software coding canreadily be prepared by skilled programmers based on the teachings of thepresent disclosure, as will be apparent to those skilled in the softwareart.

In some embodiments, the present invention includes a computer programproduct which is a storage medium or computer readable medium (media)having instructions stored thereon/in which can be used to program acomputer to perform any of the processes of the present invention. Thestorage medium can include, but is not limited to, any type of diskincluding floppy disks, optical discs, DVD, CD-ROMs, microdrive, andmagneto-optical disks, ROMs, RAMs, EPROMs, EEPROMs, DRAMs, VRAMs, flashmemory devices, magnetic or optical cards, nanosystems (includingmolecular memory ICs), or any type of media or device suitable forstoring instructions and/or data.

The foregoing description of the present invention has been provided forthe purposes of illustration and description. It is not intended to beexhaustive or to limit the invention to the precise forms disclosed.Many modifications and variations will be apparent to the practitionerskilled in the art. The modifications and variations include anyrelevant combination of the disclosed features. The embodiments werechosen and described in order to best explain the principles of theinvention and its practical application, thereby enabling others skilledin the art to understand the invention for various embodiments and withvarious modifications that are suited to the particular usecontemplated. It is intended that the scope of the invention be definedby the following claims and their equivalents.

What is claimed is:
 1. A method for testing an event processing system,comprising: sending one or more input events in an event stream to anevent processing system, wherein each said input event is associatedwith a timestamp; receiving one or more actual output events from theevent processing system, wherein each said actual output event isassociated with a timestamp; and comparing said one or more actualoutput events in an event window with one or more expected output eventsfor the event window.
 2. The method of claim 1, further comprising:organizing one or more tests in a tree structure.
 3. The method of claim1, further comprising: allowing each said event to be one of a classbased event and a map based event, wherein each said event is associatedwith one or more pseudo properties, which are used for input andverification.
 4. The method of claim 1, further comprising: allowing theevent stream to be one of an application timestamped stream and a systemtimestamped stream.
 5. The method of claim 1, further comprising:allowing at least one event in the event stream to be a temporal event.6. The method of claim 1, further comprising: declaring an input data,an output data, and a processing rule of a test in an XML file, whereinthe XML file is associated with an XML schema that is designed for thetesting event processing system.
 7. The method of claim 6, furthercomprising: allowing each said test to include a plurality of asserts,wherein each assert defines one or more input events and an expectedoutput event.
 8. The method of claim 6, further comprising: allowingdata sharing among different tests.
 9. The method of claim 6, furthercomprising: using an individual verification strategy to verify eachsaid assert based on the one or more actual output events and theexpected output event.
 10. The method of claim 9, further comprising:allowing the individual verification strategy to be based on at leastone of input-output verification, stepped input-output verification,event-by-event verification, prepared data in event window, and multiplestreams input.
 11. A system for handling various database statenotifications in a transactional middleware machine environment,comprising: one or more microprocessors; a testing framework, running onthe one or more microprocessors, wherein the testing framework operatesto send one or more input events in an event stream to an eventprocessing system, wherein each said input event is associated with atimestamp; receive one or more actual output events from the eventprocessing system, wherein each said actual output event is associatedwith a timestamp; and compare said one or more actual output events inan event window with one or more expected output events for the eventwindow.
 12. The system according to claim 11, wherein: one or more testsare organized in a tree structure.
 13. The system according to claim 11,wherein: each said event is one of a class based event and a map basedevent, wherein each said event is associated with one or more pseudoproperties, which are used for input and verification.
 14. The systemaccording to claim 11, wherein: the event stream is one of anapplication timestamped stream and a system timestamped stream.
 15. Thesystem according to claim 11, wherein: at least one event in the eventstream is a temporal event.
 16. The system according to claim 11,further comprising: an XML file, which declares an input data, an outputdata, and a processing rule of a test, wherein the XML file isassociated with an XML schema that is designed for the testing eventprocessing system.
 17. The system according to claim 16, wherein: eachsaid test includes a plurality of asserts, wherein each assert definesone or more input events and an expected output event.
 18. The systemaccording to claim 16, wherein: the testing framework operates to sharedata among different tests.
 19. The system according to claim 16,wherein: the testing framework operates to use an individualverification strategy to verify each said assert based on the one ormore actual output events and the expected output event, wherein theindividual verification strategy is based on at least one ofinput-output verification, stepped input-output verification,event-by-event verification, prepared data in event window, and multiplestreams input.
 20. A non-transitory machine readable storage mediumhaving instructions stored thereon that when executed cause a system toperform the steps comprising: sending one or more input events in anevent stream to an event processing system, wherein each said inputevent is associated with a timestamp; receiving one or more actualoutput events from the event processing system, wherein each said actualoutput event is associated with a timestamp; and comparing said one ormore actual output events in an event window with one or more expectedoutput events for the event window.